Long-term properties of time series generated by a perceptron with various transfer 

functions 
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We study the effect of various transfer functions on the properties of a time series generated 
by a continuous-valued feed-forward network in which the next input vector is determined from 
past output values. The parameter space for monotonic and non-monotonic transfer functions is 
analyzed in the unstable regions with the following main finding; non-monotonic functions can 
produce robust chaos whereas monotonic functions generate fragile chaos only. In the case of non- 
monotonic functions, the number of positive Lyapunov exponents increases as a function of one of 
the free parameters in the model, hence, high dimensional chaotic attractors can be generated. We 
extend the analysis to a combination of monotonic and non-monotonic functions. 
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I. INTRODUCTION 

One of the developing subjects in the research of neu- 
ral networks is the analysis of time series. There are 
several approaches in this field such as prediction, char- 
acterization, modeling etc. In this paper, we focus on 
understanding the interplay between the type of trans- 
fer function used and some quantitative measures of the 
time series generated. In particular we are interested in 
the classification of the possible types of sequences gener- 
ated by the network and their characteristics according 
to the nature of the attractor of the dynamics. Previ- 
ous analytical studies concentrated on the stable regime 
of the parameter space of feed-forward networks with a 
feedback loop that generate time series One of 

the main questions we address in this paper regards the 
behavior of the system in the unstable regime and how 
varying the transfer function affects the asymptotic be- 
havior of the sequence generated by the model. 

In order to characterize the dynamical system we ana- 
lyze its properties while varying some control parameters. 
Analysis of the parameter space of a map enables us to 
classify the type of flow in phase space in the vicinity 
of a given vector of parameters. An interesting question 
which arises is whether it is possible to generate high di- 
mensional attractors and control their properties, e.g. , 
the attractor dimension (a global property of the phase 
space) and the robustness (a local property of the param- 
eter space). 

As we shall see, there exists a clear distinction between 
monotonic and non-monotonic transfer functions, e.g. in 
terms of the structure of parameter space and attractor 
dimension. We shall try to illuminate this phenomenon 
as well as its relation to the possibility of generating ro- 
bust chaos. The concept of robust chaos (see [Q) is asso- 
ciated with an attractor for which the number of positive 
Lyapunov exponents (in a region of parameter space) is 
larger than the number of free (accessible) parameters in 
the model. Moreover, in the vicinity of a chaotic param- 
eter's vector, no periodic attractors are found. We give 



strong indications to support our conjecture which states 
that monotonic functions are not capable of generating 
robust chaos while non-monotonic functions are. 

The analysis presented in this paper is performed 
mostly for a perceptron with weights composed of a sin- 
gle Fourier component with an additional bias term, ex- 
cept where otherwise mentioned. There are two main 
reasons behind this choice. First, this choice leads to a 
two dimensional parameter space (/?, 6), gain and bias re- 
spectively, which can be conveniently visualized, whereas 
larger parameter space is more difficult to handle. Sec- 
ond, all the important characteristics are already mani- 
fested in this case. The existence of a bias in the weights 
is important for producing unstable dynamics in the case 
of monotonic functions, therefore it is crucial to include 
this parameter. 

The paper is organized as follows. In section 0, the 
model is described and some of the relevant results pre- 
viously obtained are reviewed. In section ^ the class 
of monotonic functions, such as the hyperbolic-tangent 
function, is analyzed. The cases of two and three input 
units, N = 2, 3 , are examined and compared to previous 
findings. We analyze the parameter space using numer- 
ical methods, e.g. calculating Lyapunov spectrum, at- 
tractor dimension (see Appendix), to identify stable and 
chaotic regions. The main conclusion is that the model 
with monotonic functions is indeed rather stable in the 
sense that even in regimes where chaotic behavior can be 
found, the chaos is fragile and small variations of the pa- 
rameters drive the system to stable dynamics. In section 
IV non-monotonic functions are examined both analyt- 
ically and numerically. The issue of high dimensional 
attractors is treated as well as the structure of parame- 
ter space and the possibility of generating robust chaos. 
Finally, in section [v| we discuss the case of a transfer 
function which is a combination of monotonic and non- 
monotonic functions. The appendix contains the techni- 
cal details concerning our analysis of the parameter space 
which one should refer to while reading sections III and 
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II. THE MODEL 

Let us consider a perceptron with iV input units and 
weights W. For a given input vector at time step t, S* 
(Spj = 1, . . . , N), the network's output S* ut is given by 

where (3 is a gain parameter and / is a transfer function. 
The input vector at time t + 1 is defined by the previous 
output values in the following dynamic rule: 

S\ + = S* ut ; Sj + — Sj_ 1 j — 2, . . . , N (2) 

Since the network generates an infinite sequence from 
an initial state, this model was denoted in previous pa- 
pers as a Sequence- Generator (SGen) e.g. ||. We restrict 
the discussion to bounded, symmetric nonlinear transfer 
functions, i.e. 

/ :K^1Z , \f(x)\ < oo VxeK. (3) 

A prescription for the weights is given by the following 
form 

Wj = ^2a p cos(^k p j + 7T0 p ) + b 
p 

j = l...N, (4) 

where {a p } are constant amplitudes; {k p } are positive 
integers denoting the wave numbers; b is the bias term 
and p runs over the number of Fourier components com- 
posing the weights. In the following we investigate only 
the cases p = 1 or 2 in order to keep the dimension of the 
parameter space as small as possible. 

Our main concern is the differences imposed by the 
transfer function on the asymptotic behavior of the 
time series generated by the model. We concentrate on 
two classes of functions, monotonic and non-monotonic, 
which are exemplified in detail by hyperbolic-tangent and 
sin functions respectively. This model was analyzed in 
various cases, all of them in the stable regime, for mod- 
erate values of the gain parameter/3. The other extreme, 
[3 — » oo was also treated (see We concentrate 

on the intermediate regime for which unstable behavior 
emerges. 

Let us review the relevant results previously obtained. 
In the case of a 'perceptron-SGen' with general weights 
and an odd transfer function, the system undergoes a 
Hopf bifurcation at some critical value of the gain pa- 
rameter. The stationary solution above the bifurcation 
value is characterized by a quasi-periodic attractor flow 
governed by one of the Fourier components of the power 
spectrum of the weights, hence the attractor dimension 
(AD) is typically one. This type of flow becomes un- 
stable at higher gain value, this being the focus of this 



paper. We should point out here that there are cases 
for which a stable two-dimensional (2D) attractor is ob- 
served ||, nevertheless their measure is zero. The results 
were extended to Multi-Layer Networks in j|J^] where the 
attractor dimension, in the stable regime, is found to be 
bounded in the generic case by the number of hidden 
units, independent of the complexity of the weight vec- 
tors. 



III. MONOTONIC FUNCTIONS 

In this section we discuss the case of monotonic trans- 
fer functions. This family of functions is typical to neural 
networks for several reasons, one of which is their biolog- 
ical plausibility (see e.g. ||). For the rest of this section 
we use the hyperbolic-tangent function as being repre- 
sentative; however, the main results are common to other 
monotonic transfer functions. The output, S ou t, for this 
case is given by 

N 

SL = tanh(/?]TT4^) (5) 
j=i 

The weights consist of a single biased Fourier component 
as follows 

Wj = acos(— kj + ■*</)) + b j = l...N, [-1,1] 

(6) 

In the following we set a = 1 to reduce the dimensionality 
of the parameter space. For the simplest case of two 
inputs, N = 2, the equation which describes this map is 
simply: 

S t+1 = tanh [PiWxS 1 + W 2 S t - 1 )] 

where Wi i = 1, 2 are given by Eq. (g). The special case 
0=1 reduces to 

S t+1 = tanh [/3 (S*(l + b) + SP'^-l + 6))] (7) 

which is equivalent to a physical model of a magnetic 
system, ANNNI model @,|lj, that was intensively in- 
vestigated in the past. This map is capable of generating 
stable attractors (nontrivial fixed points, periodic and 
quasi-periodic orbits) as well as unstable chaotic behav- 
ior. The commensurate phase of the map is presented in 
ifiof and therefore will be omitted here. 

The goal of the analysis of the parameter space is to 
classify the dynamics in a region of parameter space. The 
analytical part is unfortunately absent here due to the 
limitations set by the transfer function. It transpires 
that the class of monotonic functions does not give rise to 
critical points since the first order derivatives of the map 
are always positive, therefore the determinant is bounded 
away from zero. For large periodic orbits, however, it can 
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come very close to zero, therefore the structure can be 
somewhat more similar to maps that do contain critical 
points (see discussion in section IV). Therefore, let us 
turn to a numerical analysis. In order to answer ques- 
tions such as the existence of a robust chaos, the param- 
eter space is sampled in a high resolution, up to 1CP J 
in each direction. Figure |l| depicts a section of the pa- 
rameter space for the case <p = 1. The black area leads 
to chaotic behavior with one positive Lyapunov expo- 
nent. This area is not a compactly dense unstable re- 
gion. In fact each unstable point has stable neighbors 
which lead to periodic attractors. The remaining space 
in this region gives rise to stable attractors. The cen- 
ter of the bold circle represents the point mentioned in 
fiof which leads to chaotic behavior with the parameter 
value [/3 = 4.24155, b = 0.17881]. A perusal of the figure 
shows that the unstable region is constructed around a 
ID curve. For other choices of the phase <\> the param- 
eters of this curve change, not its nature. Moreover, we 
note that the unstable points lead to a mixed behavior, 
i.e. both stable and unstable behavior can be obtained 
from the same vector of parameters, depending on initial 
conditions. The dimension of the chaotic attractors in 
this region was calculated using the Kaplan- Yorke con- 
jecture 117 (see Appendix) and presented in the insert 
of Fig. U"^The figure is a projection of the 3 variables, 
AD,f3,b, on the AD-/3 plane, i.e. for each value of the 
gain (3 all the unstable points along the b axis are pre- 
sented. It is clear that the dimension is typically between 
1.0 and 1.3 

The same analysis was applied to the case N = 3 
(which is similar to the dynamics of the ANNNI model 
with competing interactions between third neighbors 
along the axial direction Figure || presents the re- 

sults of the same analysis which was applied for N = 2. 
The insert shows the continuation of the figure for higher 
gain values indicating that the unstable behavior can be 
found for any small bias values. The reason for taking 
4> = 1 here as well, originates from the fact that larger 
phases tend to generate more unstable regions. A sim- 
ilar behavior was found in the case of a binary output 
(3 — > oo) where the size of the cycles increases with <fi 
. It was found that the phase diagram is basically the 
same as for N = 2, i.e. the main features as described 
above are also present here. 

The general case (j) < 1 can be analyzed in the same 
manner as that described above. We note that for N = 2 
no unstable regions are found for <f> < ^ 



In higher dimensions (larger N) it is necessary to use 
more Fourier components to describe general weights, 
therefore more parameters are required - amplitudes and 
phases. As before, we restrict the dimension of parame- 
ter space to two. Figure || depicts a region in parameter 
space for the case N = 9 with = 1 (unstable behav- 
ior is found outside this region as well). The unstable 
points cover a significant part of the space. Qualitatively, 
the parameter space is similar to that of N — 2, 3 in 



the sense that the chaotic regions are mixed and fragile. 
However, as N increases the structure of the parame- 
ter space becomes more involved as larger cycles become 
available. Moreira and Salinas jl2| have already men- 
tioned that such a complication is expected at larger (3 
in their model (N — 3). We should stress here that the 
apparent dense regions of unstable points do not imply 
robust chaos since all the characteristics discussed previ- 
ously for smaller systems are present here, namely there 
is only a single positive Lyapunov exponent; the chaotic 
regions are fragile, i.e. in the vicinity of every unsta- 
ble point there exists a stable one. In particular, these 
points generate a mixed behavior in phase space. Stable 
and unstable attractors are possible, depending on the 
initial conditions, hence both have a non- vanishing basin 
of attractions. 

The examples provided so far consist of weights with a 
single Fourier component. Nevertheless, we do not expect 
any significant quantitative changes in the cases where 
the weights consist of more Fourier components, besides 
the obvious addition of free parameters. The reason is 
that the number of positive Lyapunov exponents does 
not increase. For conciseness, we tested the case of two 
Fourier components with bias, p — 2 in Eq. (|^). A few 
cases with arbitrary amplitudes and phases were chosen. 
The results indicate that our conclusions are applicable 
in the more general case. 

In all our simulations we found no regions with more 
than a single positive exponent (for N up to 60), includ- 
ing many cases with randomly chosen weights. Therefore, 
we conjecture that the SGen with hyperbolic-tangent 
function typically exhibits unstable behavior with a sin- 
gle positive Lyapunov exponent. 

We conclude with the observation that the bias term 
6, Eq. ([!]) is crucial for producing chaotic behavior in 
the model with a monotonic transfer function. Another 
important ingredient is the existence of a large enough 
phase, at least when the weights consist of a single 
Fourier component. It is possible that additional Fourier 
components are sufficient to generate unstable behavior 
(without large phase), however, larger phase significantly 
increases the number of unstable points. 



IV. NON-MONOTONIC FUNCTIONS 

Applying a non-monotonic transfer function dramati- 
cally alters the structure of parameter space with respect 
to monotonic functions. One is able to observe robust 
chaos, and the possible number of positive exponents is 
no longer bounded by one. In the following analysis we 
treat the class of odd non-monotonic functions and use 
the 'sin' as a representative function. 

As mentioned in section [n], quasi-periodic stationary 
solutions were found analytically for odd functions which 
are valid below some critical value of the gain parameter. 
In this section we focus on the region beyond that value. 
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Note that in contrast to monotonic functions, unstable 
dynamics can be obtained with phase and bias equal to 
zero. Indeed, in the sequel we use <j> = and only two 
dimensional parameter space, (3-b, as in the previous sec- 
tion. 

Before we turn to the analysis of the parameter space, 
let us demonstrate a manifestation of a chaotic behavior 
for concrete parameter values in small networks, N — 
3, 4 , via the mechanism of period doubling. The dynamic 
system is described by Eq. (||) where the output value is 
given by Eq. (Q) with / = sin transfer function. Figure 
presents a sequence of bifurcations of the output values 
(denoted by "amplitude") on a limit cycle as a function 
of the gain (3 for N = 3. For clarity we plot the sequence 
originating from one branch. 

Figure ra presents the difference between the values of 
(3 at which successive period doubling occurs 



A„ — Pn+l — fin 



(8) 



This figure depicts three cases: N — 3, N — 4 with 
weights consisting of a single Fourier component and 
N = 4 with arbitrary weights. Although the running 
index n starts from 1, the actual number of bifurcations 
is somewhat higher. Clearly the difference A„ is an ex- 
ponential decreasing function of the form 



A„ 



(9) 



The constant 5 was evaluated from the slope for the three 
cases and found to be in good agreement with Feigen- 
baum's universal constant (~ 4.669) JTif . 

The analytical analysis of the parameter space con- 
centrates on obtaining the spine loci of a given map. 
The spine locus is associated with the parameter vec- 
tors which lead to a super-stable attractor (e.g. in one 
dimension, the first order derivatives of the map vanish 
at the super-stable attractor). Barreto et. al. |J have 
conjectured that the structure of the parameter space is 
determined primarily by the location and dimension of 
the spine loci. A window is constructed around the spine 
locus which leads to a stable attractor. Generally speak- 
ing, the window is called 'limited' if the spine locus is 
an isolated point in parameter space, whereas it is called 
'extended' if the spine is of higher dimension. 

We turn now to a more systematic investigation of 
the parameter space, starting with the simplest case of 
N = 2. Since the number of free parameters is equal 
to the size of the system, the number of positive expo- 
nents is at most the number of parameters, therefore one 
should not expect a robust chaos (unless fixing one of the 
parameters). The weights for a single Fourier component 
with a = 1 and (j> = are given by Eq. (|^) . In principle 
we could take <j) ^ which may drive a fraction of the 
periodic orbits to quasi-periodic ones. 

Similarly to Eq. (@), the map can be written as follows 



with the following fixed point (f.p.) 

S* = em(20bS*) 



(11) 



The stability of the f.p. can be analyzed from its corre- 
sponding Jacobian matrix 



M = 



[3{-l + b) cos(2/3bS*) [3(1 + b) cos(2/365*) 
1 



(12) 



In order to identify the spine locus, the following con- 
ditions must be satisfied: det M = trM = (Ai = 
A2 = 0). It turns out that both constraints on the eigen- 
values give rise to the same condition, cos(2/36S'*) = 0. 
Therefore, we can say that the constraints are degener- 
ate. Combining this condition with Eq. ( pd| ) we obtain 
S* and the relation 



(3b=-(2n + l) 



0,1, 



(13) 



This equation holds for b > 0. For b < there are no 
f.p. solutions that satisfy the constraint. The main spine 
for b < is related to a 2-cycle solution which can be 
obtained numerically. Other f.p's which do not meet the 
constraint are possible and belong to a different curve in 
parameter space. 

In a similar way, the constraints of the 2-cycle ( S t+2 — 
S* ) spine locus gives the following relation between (3 and 
b 



C 



sin(/3 [( 
(An 



-l + 6) + (l- 
]>- 2/3(1 



b)C] 
■b) 



2(3(-l + b) 



C 



n = 0,±l,. 



(14) 



S t+1 = sin [13 ((-I + 6)5* + (1 + bjS'- 1 )] 



(10) 



Turning to numerical analysis, Fig. g depicts a region 
of parameter space where areas that lead to chaos are 
marked. This region was sampled exhaustively in a res- 
olution of « 10~ 5 in each direction. Several random 
initial conditions were used for each parameter value to 
avoid isolated cycles. The dark area corresponds to a re- 
gion with one positive exponent while the gray area cor- 
responds to a region with two positive exponents. The 
dashed line is the calculated spine locus of the f.p. defined 
by Eq. (1^) for the first branch, n — 0. Let us discuss 
briefly the structure of the parameter space. The dark 
region (left-hand-side of the figure) contains extended 
(stable) windows (embedded white areas) associated with 
cycles of different length. The common feature of these 
windows is the fact that they are surrounded by unsta- 
ble regions with one positive exponent. As we move to 
the right-hand-side of the figure, a region with two posi- 
tive exponents emerges. The spine locus depicted enters 
this region since, as mentioned above, we started from 
several initial conditions, therefore the dynamics is typ- 
ically attracted to unstable cycles of higher order. In 
order to isolate the spine of the f.p. and the 2-cycle from 
higher order cycles, we analyze the window when the ini- 
tial condition is fixed to Si = 1. Figure depicts a region 
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of parameter space for which this analysis was applied. 
In this case, the basic nature of these spines is revealed 
and a clear extended window is constructed around the 
solid curve (f.p., Eq. (|l3|) ) as well as the solid curve with 
circles (2-cycle, Eq. ([[W). 

Analysis of the case N = 3 is similar to N = 2. The 
spine locus for the f.p. is given by 

(2n + 1W 

(3b =- 6>0 , n = 0,l,... (15) 

We can generalize the equation for the spine locus of the 
f.p. for any N 

(2n + l)vr , , 

(3bN=- b>0 ,n = 0,l,... (16) 

The constraint for the 2-cycle is obtained from similar 
conditions formulated for N = 2. The relation between 
(3 and b is the following 



C 



sin(/?[(l/2 + 2&) + (-l/2 + &)Cl) = C 
(An + 1)tt + /3(1 - 26) 



(3(1 + 46) 



0,±1,. 



(17) 



The case N = 3 reveals another aspect in the structure 
of parameter space. There are regions for which we find 
three positive Lyapunov exponents. In such regions, we 
observed a robust chaos, namely, small changes of the 
parameters would not destroy the chaotic behavior. 

In principle we can construct the conditions of the 
spine locus for larger cycles and larger systems, however 
the task becomes much more involved as the cycle length 
increases. 

Let us now extend our analysis for large systems. Two 
questions come to the fore: 

1. Can we find regions for which the chaotic dynamics 
is robust, and how frequent are they ? 

2. Is there a simple relation between the attractor di- 
mension and the control parameters or, in other 
words, can we control the attractor dimension ? 

We saw previously that even in the case N — 3, a ro- 
bust chaos is observed. However, this type of dynamics 
is of little interest since the volume in phase space is ex- 
panding, Xi > , hence the bounded space is filled. 
The more interesting case is a motion which is confined 
to an attractor, yet the number of positive exponents is 
larger than the number of free parameters. We claim that 
the possibility to find regions with an increasing number 
of positive exponents, grows with (3. This means that we 
have a natural parameter in the model that controls the 
degree of the chaos. In addition, this parameter controls 
the dimension of the attractor. 

In order to test this hypothesis we used a larger sys- 
tem, N — 17. To convince the reader that our analysis 



is not restricted to the simple case of a single Fourier 
component, we used more complicated weights consisting 
of two Fourier components with irrational phases and a 
bias term. The amplitudes and phases of the components 
were kept fixed, therefore we have the same two dimen- 
sional parameter space, as before. The exact details of 
the amplitudes and phases are of no importance. 

A close inspection of the parameter space reveals the 
following regimes: First, the incommensurate regime 
which corresponds to the irrational phases of the weights. 
Above some value of the gain parameter (depending on 
the details of the weights) most of the space is associated 
with chaotic dynamics. The number of positive expo- 
nents in this regime grows as j3 increases until the sum 
of the exponents becomes positive. In this regime, we 
observed a relatively monotonic growth in the attrac- 
tor dimension, calculated using Eq. ( [jl"l) (see Appendix). 
Figure [8] depicts the attractor dimension for a fixed bias 
value. Clearly, the dimension grows monotonically. The 
figure also shows the number of positive exponents which 
grows with 0. Each point was averaged over 10 random 
initial conditions in order to check whether the same at- 
tractor is sampled. Indeed, the errors are less than 1% 
and typically much less, therefore they are not presented. 
(Note that there are cases, not shown, for which the line 
6 = const crosses a window. In such regions, the at- 
tractor dimension decreases and then continues to grow 
once the window is passed) . As the sum of the exponents 
becomes positive, the attractor dimension saturates the 
dimension of the system, N. 

Finally we validated this results using a different 
method. We tested several points in parameter space by 
estimating the attractor dimension from the time series 
generated by a network and compare it to the estima- 
tion using Eq. pl|). The time series was recorded from 
a system with the same parameters and the attractor 
dimension was calculated from the reconstructed phase 
space using the method of Correlation-Integral |lE| . The 
results confirm our hypothesis for the monotonic relation 
between the number of positive exponents and (3. 



V. COMBINATION OF MONOTONIC AND 
NON-MONOTONIC FUNCTIONS 

In this section we discuss the mixed case where the 
transfer function can be written in the following way 



f(x) = f m (x) + ef nm (x) 



(18) 



where f m t n m) represents a monotonic (non-monotonic) 
function; e is a mixing parameter (not necessarily small). 
For concreteness, assume that f m ( x ) = tanh(x) and 
fnm{x) — sin(x). Let x — (3W ■ S and the weights are 
given by Eq. (H) (taking <f> — for simplicity). Follow- 
ing the same developments shown in we develop an 
asymptotic periodic solution of the form 
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Si = tanh(Acos(— kl + B)) + e sin(A cos( — kl + B)) 

(19) 

where A = A{fi) , B = B(J3). The coefficients A, B can 
be obtained from the self-consistent equations : 

A = ipaE^DWfi/rHp!)- 2 ; B = 

B = pNbE7= 1 D(p)B 2p - 1 ((2 P )l)- 1 ; A = Q 

(20) 

where D{p) = 2 2 p(2 2 p - l)B 2p + 2ep{-iy +1 and B 2p 
are the Bernoulli numbers. 

As described in Q, when the gain value increases the 
system undergoes a transition from the trivial solution, 
Si = 0, to a state which is governed by one of the two 
possible attractors: a fixed point (A — , B 0) or 
a periodic solution, depending on the relation between 
Pel, Pc2 (the critical value for the onset of each attractor). 
At higher gain values (/? > /3 c i,/3 C 2) both attractors are 
stable and the system will flow to one of them, depending 
on the initial condition. When the gain parameter is 
further increased, one observes unstable dynamics of the 
type described in this paper. The mixing parameter e 
controls the actual point from which the parameter space 
is governed by the non-monotonic function. 

Finally, we note that it is easy to generalize this solu- 
tion for tf> 7^ and weights which contain more Fourier 
components (Eq. (Q) following 

VI. DISCUSSION 

In this paper we analyzed a class of neural networks 
in the context of time series generation and focused on 
the effect of the transfer function on long-term behavior. 
We suggested a natural way to classify transfer functions 
into monotonic and non-monotonic functions. The class 
of monotonic functions can generate chaotic dynamics, 
however the weights should contain a non-trivial bias 
term and a phase. The chaos can be regarded as fragile, 
i.e. in regions where unstable behavior can be observed, 
the parameter space is characterized by points around 
which the parameter vectors lead to a stable dynamics 
(limited windows). As the size of the network increases, 
the structure of parameter space becomes more involved 
due to the appearance of longer cycles. On the other 
hand, the class of non-monotonic functions is capable 
of generating robust high dimensional (chaotic) attrac- 
tors. This means that there exist regions of parameter 
space for which slight changes in the vector of accessi- 
ble parameters will not stabilize the system. Although 
the analysis presented in the paper was exemplified by 
an odd non-monotonic function, the results we obtain, 
which are mainly related to the regime of (3 values where 
unstable behavior emerges, include even functions as well 
II- 



One must refer to another paper Jl6| that focuses on 
searching robust chaos in recurrent neural networks using 
weight space exploration. Their motivation and meth- 
ods differed from ours. However, their main result which 
states that robust chaos can be achieved using a non- 
monotonic transfer function only, is in agreement with 
our conclusions. 

Another aspect characterizing non-monotonic func- 
tions is the potential to monotonically increase the at- 
tractor dimension over a broad range of parameters. This 
interesting effect is achieved by increasing the gain pa- 
rameter, (3. Unless the path of the vector of parameters 
intersects the boundary of a spine locus, the attractor 
dimension increases monotonically with until it satu- 
rates the dimension of the system. From this point, one 
can no longer define the dynamics as an attractor since 
a volume of phase space expands. 

Based on these results we can formulate the follow- 
ing conclusion: A 'perceptron-SGen' with non-monotonic 
transfer function can generate a chaotic attractor much 
more easily than with a monotonic function. In addition, 
when the chaos is robust one can expect the learning 
process to be easier since the extended parameter space 
which includes the attractor dimension is smooth in these 
regions. 

We further showed that the results can be extended to 
transfer functions which are combinations of monotonic 
and non-monotonic functions. The asymptotic stable at- 
tractor is developed similarly to the pure case of mono- 
tonic / non-monotonic functions. The structure of the 
unstable region is governed by the function that looses 
its stability in that region. There are three possible sce- 
narios: either one of the two types of functions loses its 
stability while the other remains stable or both functions 
lose their stability. The first two cases are actually cov- 
ered throughout the paper. 

The extension of the results to the case of Multi-Layer 
Networks raises new interesting questions, e.g. to what 
extent our findings remain valid; how does the combined 
perceptron-SGen's affect each other, namely, do they act 
to stabilize the dynamics or does a chaotic unit maintain 
its behavior, thus reflecting its robust nature. Another 
issue of major importance is the attractor dimension of 
the MLN. While a 'perceptron-SGen' can typically gen- 
erate a ID attractor in the stable regime, we saw that 
applying a non-monotonic transfer function to this net- 
work can generate much higher and continuous attractor 
dimension. On the other hand, a 'MLN-SGen' can gen- 
erate higher (integer) attractor dimension in the stable 
regime (see (^H). Do 'MLN-SGen's generate continu- 
ous high dimensional attractors as well ? 
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APPENDIX 

In order to characterize the parameter space numer- 
ically, we apply a rather straightforward method. For 
a dense mesh of points in the j3-b plane we calculate the 
spectrum of Lyapunov exponents associated with the dy- 
namics, from which we can obtain the stability of the 
attractor and its attractor dimension. This procedure is 
applied after the transient and for several randomly cho- 
sen initial conditions. The spectrum is estimated using 
an algorithm suggested by Wolf et. al. p8| . Basically, the 
algorithm evolves an orthonormal basis by multiplying 
each vector with the Jacobian matrix which is evaluated 
along the trajectory. To overcome the problem of expo- 
nential decreasing of the vectors associated with smaller 
eigenvalues, the principal vectors are re-orthonormalizcd 
frequently. This procedure ensures that our analysis does 
not run into roundoff errors. This algorithm measures 
the average exponential change of a volume along the 
trajectory in state-space, using the rate of change in the 
principal vectors. 

The attractor dimension is estimated using the 
Kaplan- Yorke conjecture Il7|] that gives the following re- 
lation between the (sortedjspectrum of Lyapunov expo- 
nents Xi (Ai > A2 . . .) and the attractor dimension dk y 
(information dimension) 

l A n+l| 

where n is defined by the condition JY =1 ^* ^ an( ^ 

EILi^<o. 

We are aware of a problem associated with the Kaplan- 
Yorke conjecture. In cases where the spectrum has some 
very small exponents it is possible to obtain a biased 
estimation of the dimension since the sum described in 
Eq. (|2l]) may fluctuate around zero due to one of the 
exponents. In such cases, we take additional precaution 
to avoid this problem. 
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FIG. 1. Analysis of a region in parameter space for hyper- 
bolic-tangent transfer function and N = 2. Points that lead 
to chaotic trajectories are marked. The remaining space in 
this region leads to stable attractors. The center of the circle 
at (4.24, 0.178) represents the chaotic point discussed in |l(]] 
. Insert: the attractor dimension (AD) of the chaotic points 
shown in this region. For each value of the parameter /3, the 
AD of all the chaotic points along the b axis are drawn. 
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FIG. 3. Example of a region in parameter space for hyper- 
bolic-tangent transfer function and N = 9 where points that 
lead to chaotic trajectories are marked. The remaining space 
in this region leads to stable attractors. 
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FIG. 4. A sequence of period doubling bifurcations for a 
network with a sin transfer function and N = 3. The weights 
consist of a single Fourier component without phase and bias. 
The vertical axis, denoted by 'amp', is the actual amplitude 
value of the cycle in phase space. Insert: a blowup of the 
pointed region. 
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FIG. 2. A region in parameter space for hyperbolic-tangent 
transfer function and TV = 3. Points that lead to chaotic 
trajectories are marked. The remaining space in this region 
leads to stable attractors. Insert: A continuation of the main 
figure for smaller values of b in a log-log plot. 
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FIG. 5. The difference A n (Eq. (g)) for three cases. The 
lines are exponential fit of the data. The solid line represents 
the fit for TV = 3 with a slope of 4.69 ± 0.05. The dashed line 
represents N — 4 with a slope of 4.66 ± 0.02 and the dotted 
line represents N = 4 with an arbitrary weights with a slope 
of 4.65 ± 0.02 . 
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FIG. 6. Analysis of a region in parameter space for a net- 
work with a sin transfer function and N = 2 where points 
that lead to chaotic trajectories are marked. The dark(gray) 
colors correspond to areas with one(two) positive exponent. 
The remaining space in this region leads to stable attractors. 
The bold dashed line is the spine locus of the f.p. defined by 
Eq. n = 0. 



FIG. 7. Analysis of a region in parameter space around 
the main spine loci for a sin transfer function and N = 2. 
The initial condition in phase space is fixed to Si = 1. The 
dark(gray) color corresponds to areas with one(two) positive 
exponents. The remaining space in this region leads to stable 
attractors. The solid line is the spine locus of the f.p. defined 
by Eq. (^), and the solid line with circles is the calculated 
spine locus of the 2-cycle attractor defined by Eq. (|l4|). 
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FIG. 8. The attractor dimension (AD) as a function of the 
gain for a sin transfer function where TV = 17, b = and 
<j> — 0. The solid curve is the AD and the dashed line below 
represents the number of positive exponents. 
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